Gaze detection, identification and control method

ABSTRACT

A gaze detection, identification and control method includes steps of providing a portable electronic device, performing a continuous image capture on a user through an image capture unit of the portable electronic device to sequentially obtain an initial image and a plurality of detected images, detecting and defining a gaze position of the user according to the initial image, identifying a change amount and a change direction of the gaze position of the user according to the detected images, and sending a control command according to the change amount and the change direction. Therefore, the advantages of easy recognition and time-saving are achieved.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Taiwan Patent Application No.TW106132857, filed on Sep. 25, 2017, the entire contents of which areincorporated herein by reference for all purposes.

FIELD OF THE INVENTION

The present invention relates to an identification and detection method,and more particularly to a gaze detection, identification and controlmethod implemented by recording and tracing eyeballs and head shape.

BACKGROUND OF THE INVENTION

In the common smartphones and tablet PCs, which are the most popular onthe market, most applications are controlled through the touch of handas a control command input. Although the smartphones and the tablet PCsequipped with large screens are good for reading, it is difficult andinconvenient for the normal user to operate with merely one hand.

For example, when shooting with a camera on a smartphone or a tablet PC,to change the focus depth, the same scene must be shot through twolenses and different focusing must be performed in the process. Finally,the smartphone or the tablet PC is manually clicked to make images withshort focal length or long focal length clear. However, in a situationonly one hand of the user is available for operation, it is veryinconvenient for the user to hold the smartphone or the tablet PC andadjust the focal length or push the photographing button at the sametime. In addition, when browsing web pages, articles, letters, maps andphotos, the user must constantly slide the hand on the display to scrollthe display for reading. However, with only one hand operation, it isdifficult for the user to simultaneously hold the hand-held smartphoneor tablet PC while sliding on the display. In short, current smartphonesand tablet PCs are not friendly enough to operate with one hand. It isdifficult for the user to operate with only one hand. Therefore, afterinvestigation, research and development, a manner of control througheyeballs of human is put forward.

In the prior art, the conventional technique of control through eyeballsof human usually sets two coordinates, one is the coordinate of theeyeball, and the other is the coordinate of the cursor on the display.By capturing the coordinate positions of the pupil through the camerabefore and after the movement, the distance of the movement is enlargedand corresponded to the coordinates on the display. However, since thepupil occupys a zone of low-proportion of the eye, a precise correctionmethod must be applied, and re-corrections are required for differentusers. It not only causes problems of recognition, but also causes wasteof time and other invisible costs.

Therefore, there is a need of providing a gaze detection, identificationand control method distinct from the prior art in order to solve theabove drawbacks, achieve precise recognition and implement diversefunctions.

SUMMARY OF THE INVENTION

Some embodiments of the present invention are to provide a gazedetection, identification and control method in order to overcome atleast one of the above-mentioned drawbacks encountered by the priorarts.

The present invention provides a gaze detection, identification andcontrol method. By continuously detecting and identifying the gazingposition of the user for sending the control command, the gaze controlmethod of controlling the portable electronic device is effectivelyprovided. Meanwhile, since the identification of the change amount andthe change direction of the gaze position of different users may beimplemented without high-precision corrections, the advantages of easyrecognition and time-saving are achieved.

In accordance with an aspect of the present invention, there is provideda gaze detection, identification and control method. The gaze detection,identification and control method includes steps of providing a portableelectronic device, performing a continuous image capture on a userthrough an image capture unit of the portable electronic device tosequentially obtain an initial image and a plurality of detected images,detecting and defining a gaze position of the user according to theinitial image, identifying a change amount and a change direction of thegaze position of the user according to the detected images, and sendinga control command according to the change amount and the changedirection

The above contents of the present invention will become more readilyapparent to those ordinarily skilled in the art after reviewing thefollowing detailed description and accompanying drawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates the flow chart of a gaze detection,identification and control method according to an embodiment of thepresent invention;

FIG. 2 schematically illustrates the configuration of a portableelectronic device applied to the gaze detection, identification andcontrol method of the present invention;

FIG. 3 schematically illustrates the detailed flow chart of a gazedetection, identification and control method according to an embodimentof the present invention;

FIG. 4A schematically illustrates the initial image captured through thegaze detection, identification and control method of the presentinvention;

FIG. 4B schematically illustrates the detected image after a movement ofthe eyeball of the user shown in FIG. 4A;

FIG. 5 schematically illustrates the detailed flow chart of a gazedetection, identification and control method according to an embodimentof the present invention;

FIG. 6 schematically illustrates the detected images captured throughthe gaze detection, identification and control method of the presentinvention;

FIG. 7 schematically illustrates the detailed flow chart of a gazedetection, identification and control method according to an embodimentof the present invention; and

FIG. 8 schematically illustrates the flow chart of a gaze detection,identification and control method according to another embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention will now be described more specifically withreference to the following embodiments. It is to be noted that thefollowing descriptions of preferred embodiments of this invention arepresented herein for purpose of illustration and description only. It isnot intended to be exhaustive or to be limited to the precise formdisclosed.

Please refer to FIG. 1. FIG. 1 schematically illustrates the flow chartof a gaze detection, identification and control method according to anembodiment of the present invention. As shown in FIG. 1, an embodimentof a gaze detection, identification and control method includes steps asfollowing. First of all, as shown in step S100, providing a portableelectronic device, in which the portable electronic device may be asmartphone or a tablet PC, but not limited herein. Next, as shown instep S200, performing a continuous image capture on a user through animage capture unit of the portable electronic device to sequentiallyobtain an initial image and a plurality of detected images. Then, asshown in step S300, detecting and defining a gaze position of the useraccording to the initial image. Next, as shown in step S400, identifyinga change amount and a change direction of the gaze position of the useraccording to the detected images. Next, as shown in step S500, sending acontrol command according to the change amount and the change direction.The control command is directly sent to a main processor of the portableelectronic device, so that an application is executed, or a specifiedfunction of the portable electronic device is started. Therefore, thegaze control method of controlling the portable electronic device iseffectively provided. Meanwhile, since the identification of the changeamount and the change direction of the gaze position of different usersmay be implemented without corrections of high precision, the advantagesof easy recognition and time-saving are achieved.

Please refer to FIG. 2. FIG. 2 schematically illustrates theconfiguration of a portable electronic device applied to the gazedetection, identification and control method of the present invention.As shown in

FIG. 2, a portable electronic device 1 applied to the gaze detection,identification and control method of the present invention includes amain processor 10, an image signal processor (ISP) 11, an image captureunit 12, a secondary image capture unit 13, a light-emitting unit 14 anda power management integrated circuit (PMIC) 15. In some embodiments,the main processor 10 may be a central processing unit, a microprocessoror a microcontroller, but not limited herein. The image signal processor11 is connected with the main processor 10 or integrated within the mainprocessor 10 for processing the captured images and then providing tothe main processor 10. The image capture unit 12 and the secondary imagecapture unit 13 are connected with the image signal processor 11 forcapturing the initial images and the detected images mentioned in theabove embodiments, but not limited herein. The light-emitting unit 14 isfor example a light-emitting diode (LED), but not limited herein. Thelight-emitting unit may be utilized as a flash light or a compensationlight. The power management integrated circuit 15 is connected with thelight-emitting unit 14 and the main processor 10. The power managementintegrated circuit 15 is controlled by the main processor 10 to providepower to the light-emitting unit 14, so that the light-emitting unit 14may perform a flash or a light compensation to meet the practicaldemands.

Please refer to FIG. 1 and FIG. 2 again. In some embodiments, theportable electronic device 1 may include only the image capture unit 12,or include the image capture unit 12 and the secondary image captureunit 13 simultaneously (e.g. image capture units having different focusdepths or different focal lengths). The step S200 of the above-mentionedembodiments is preferably implemented by the image capture unit 12, oris implemented by the image capture unit 12 and the secondary imagecapture unit 13, but not limited thereto.

Please refer to FIG. 3, FIG. 4A, FIG. 4B and FIG. 1. FIG. 3schematically illustrates the detailed flow chart of a gaze detection,identification and control method according to an embodiment of thepresent invention. FIG. 4A schematically illustrates the initial imagecaptured through the gaze detection, identification and control methodof the present invention. FIG. 4B schematically illustrates the detectedimage after a movement of the eyeball of the user shown in FIG. 4A. Asshown in FIG. 1, FIG. 3, FIG. 4A and FIG. 4B, the step S300 of the gazedetection, identification and control method of the present inventionincludes sub-steps as following. As shown in sub-step S325, judging anedge position of a head 30 of the user 3 according to the initial image2. Next, as shown in sub-step S350, defining a head shape 300 of thehead 30 of the user 3. Then, as shown in sub-step S375, calculating arelative position of eyeballs 31 of the user 3 in the head shape 300 anda distance between the eyeballs 31 and the edge position according tothe initial image 2 to define the gaze position.

For example, in a situation only the eyeballs 31 of the user 3 are movedbut not accompanying the head swing (i.e. the eyeballs 31 are moved fromthe position shown in FIG. 4A to the position shown in FIG. 4B), thegaze position and the change of the gaze position are defined accordingto the information, such like the change of the relative position andmove direction of the eyeballs 31 of the user 3 in the head shape 300.Since it is a manner of comparing the relative positions of the eyeballs31 in the head shape 300, the corrections with high precision are notrequired while utilizing the gaze detection, identification and controlmethod of the present invention. For different users, the gazedetection, identification and control method of the present inventionare directly applied, and the easy recognition is implemented. On theother hand, in a situation the head 30 and the eyeballs 31 of the user 3both are moved, the gaze detection, identification and control method ofthe present invention is preferred to calculate the distance between theeyeballs 31 and the edge position of the head shape 300. After amovement, the relative positions of the eyeballs 31 and the head shape300 will be distinct from the initial image, so that the gaze positionof the user may be judged through this manner.

Please refer to FIG. 5, FIG. 6, FIG. 1 and FIG. 4. FIG. 5 schematicallyillustrates the detailed flow chart of a gaze detection, identificationand control method according to an embodiment of the present invention.FIG. 6 schematically illustrates the detected images captured throughthe gaze detection, identification and control method of the presentinvention. As shown in FIG. 1, FIG. 4, FIG. 5 and FIG. 6, the step S400of the gaze detection, identification and control method of the presentinvention includes sub-steps as following. As shown in sub-step S433,comparing the first detected image 4 (in a time sequence) of detectedimages 4 with the initial image 2, or comparing any one of the detectedimages 4 after the second detected image 4 (in the time sequence) withthe previous detected image 4 to obtain a comparison result. In someembodiments, in the sub-step S433, the distances between the edgeposition and the top, bottom, left, right, far and near of the eyeballs31 of the user 3 are compared, but not limited herein. In someembodiments, the sub-step S433 is performed by applying a gridcoordinate system 5 on the compared detected image 4 and initial image2, or the compared two continuous detected images 4, and comparing thedifferences between coordinate positions of the eyeballs 31 of the user3 in the grid coordinate system 5. Next, as shown in sub-step S466,identifying the change amount and the change direction according to thecomparison result. In other words, the amount of the movement of theeyeballs 31 of the user 3 in the grid coordinate system 5 is the changeamount. Furthermore, when the action or the gaze of the user 3 has achange amount and a change direction compared with the previous time,the change amount and the change direction may be detected andidentified so as to control the portable electronic device through thegaze detection, identification and control method of the presentinvention.

Please refer to FIG. 7 and FIG. 1. FIG. 7 schematically illustrates thedetailed flow chart of a gaze detection, identification and controlmethod according to an embodiment of the present invention. As shown inFIG. 1 and FIG. 7, the step S500 of the gaze detection, identificationand control method of the present invention includes sub-steps asfollowing. Firstly, as shown in sub-step S533, determining an executionintensity according to the change amount and the change direction. Next,as shown in sub-step S566, sending the control command according to theexecution intensity. In other words, when the change amount of the gazeposition towards the specified change direction is relatively larger, itrepresents that the user wants to send a control command with largerexecution intensity (e.g. quickly scrolling the screen or quicklyincreasing the volume). In some embodiments, it can be set that onlywhen the change amount or the execution intensity exceeds a threshold,the control command is sent to the main processor to execute anapplication or start a specific function. Otherwise, when the changeamount or the execution intensity does not exceed the threshold, it doesnot proceed any action, but not limited herein.

The preferred embodiments of the gaze detection, identification andcontrol method of the present invention have been completely disclosed.By means of the above-mentioned embodiments, the present inventionprovides the user a control manner of the portable electronic devicewith the gaze position, thereby replacing the touch control orcooperating with the touch control. On the other hand, the gazedetection, identification and control method and the portable electronicdevice using the same of the present invention can be used to performthe following operations, but not limited thereto.

Due to the configuration of the image capture unit and the secondaryimage capture unit, the present invention can be configured to preview,photograph or browse the photos by using dual cameras. With the gazedirection of eyeballs, the focused object on the eyeball is clear andthe range outside the focused object in the photo is taken asbackground. Furthermore, when taking pictures, the image capture unitcan also be utilized to zoom in or zoom out of the focused objectthrough the change of the gaze position. Instead of using two fingers tozoom in or zoom out, the zooming can also be performed by changing thegaze position while browsing photos after photographing. When browsing a360-degree panoramic photo, the sliding of the photo for viewingdifferent parts of the photo can be performed by changing the gazeposition.

Moreover, when reading and surfing mails, articles or webpages, thedisplay, which is being read or surfed, can be stopped or scrolledaccording to the change of the gaze position. When the smartphone isringing, the incoming call can be answered or declined according to thechange of the gaze position. When playing music, the previous song, thenext song, the volume increment and the volume decrement can beperformed according to the change of the gaze position. The presentinvention can even further detect and recognize the blink of the eyeswith a variety of ways to complete more operations, but not limitedherein.

Please refer to FIG. 8. FIG. 8 schematically illustrates the flow chartof a gaze detection, identification and control method according toanother embodiment of the present invention. As shown in FIG. 8,compared with the gaze detection, identification and control methodshown in FIG. 1, a gaze detection, identification and control methodaccording to another embodiment of the present invention furtherincludes step S600 and step S700. At first, as shown in step S600,obtaining an iris feature and a head shape of the user according to theinitial image. Next, as shown in step S700, encoding the iris featureand the head shape to obtain a biometric password, and further using thebiometric password as the unlock manner or the safe certificationmanner. Since the step S100, the step S200, the step S300, the step S400and the step S500 are same as the step S100, the step S200, the stepS300, the step S400 and the step S500 of the embodiment shown in FIG. 1,they are not redundantly described herein. The step S600 and the stepS700 can be performed between the step S200 and the step S300 as shownin FIG. 8, but not limited herein. Certainly, the step S600 and the stepS700 can be performed after the initial image is obtained in the stepS200. The same advantages will also be achieved, and the sequence ofexecution is not limited.

From the above description, the present invention provides a gazedetection, identification and control method. By continuously detectingand identifying the gazing position of the user for sending the controlcommand, the gaze control method of controlling the portable electronicdevice is effectively provided. Meanwhile, since the identification ofthe change amount and the change direction of the gaze position ofdifferent users may be implemented without corrections of highprecision, the advantages of easy recognition and time-saving areachieved.

While the invention has been described in terms of what is presentlyconsidered to be the most practical and preferred embodiments, it is tobe understood that the invention needs not be limited to the disclosedembodiment. On the contrary, it is intended to cover variousmodifications and similar arrangements included within the spirit andscope of the appended claims which are to be accorded with the broadestinterpretation so as to encompass all such modifications and similarstructures.

What is claimed is:
 1. A gaze detection, identification and controlmethod, comprising steps of: (a) providing a portable electronic device;(b) performing a continuous image capture on a user through an imagecapture unit of the portable electronic device to sequentially obtain aninitial image and a plurality of detected images; (c) detecting anddefining a gaze position of the user according to the initial image; (d)identifying a change amount and a change direction of the gaze positionof the user according to the detected images; and (e) sending a controlcommand according to the change amount and the change direction.
 2. Thegaze detection, identification and control method according to claim 1,wherein the portable electronic device comprises: a main processor; animage signal processor connected with the main processor; the imagecapture unit connected with the image signal processor; a secondaryimage capture unit connected with the image signal processor; alight-emitting unit; and a power management integrated circuit connectedwith the light-emitting unit and the main processor, wherein the powermanagement integrated circuit is controlled by the main processor toprovide power to the light-emitting unit.
 3. The gaze detection,identification and control method according to claim 2, wherein the step(b) is implemented by the image capture unit, or is implemented by theimage capture unit and the secondary image capture unit.
 4. The gazedetection, identification and control method according to claim 2,wherein in the step (e), the control command is sent to the mainprocessor, so that an application is executed, or a specified functionof the portable electronic device is started.
 5. The gaze detection,identification and control method according to claim 1, wherein the step(c) comprises sub-steps of: (c1) judging an edge position of a head ofthe user according to the initial image; (c2) defining a head shape ofthe head of the user; and (c3) calculating a relative position ofeyeballs of the user in the head shape and a distance between theeyeballs and the edge position to define the gaze position, according tothe initial image.
 6. The gaze detection, identification and controlmethod according to claim 5, wherein the step (d) comprises sub-stepsof: (d1) comparing the first detected image of detected images with theinitial image, or comparing any one of the detected images after thesecond detected image with the previous detected image to obtain acomparison result; and (d2) identifying the change amount and the changedirection according to the comparison result.
 7. The gaze detection,identification and control method according to claim 6, wherein in thesub-step (d1), the distances between the edge position and the top,bottom, left, right, far and near of the eyeballs of the user arecompared.
 8. The gaze detection, identification and control methodaccording to claim 6, wherein the sub-step (d1) is performed by applyinga grid coordinate system on the compared detected image and initialimage, or the compared two continuous detected images, and comparing thedifferences between coordinate positions of the eyeballs of the user inthe grid coordinate system.
 9. The gaze detection, identification andcontrol method according to claim 1, wherein the step (e) comprisessub-steps of: (e1) determining an execution intensity according to thechange amount and the change direction; and (e2) sending the controlcommand according to the execution intensity.
 10. The gaze detection,identification and control method according to claim 1 furthercomprising steps of: (f) obtaining an iris feature and a head shape ofthe user according to the initial image; and (g) encoding the irisfeature and the head shape to obtain a biometric password.